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ABSTRACT 

In this paper, the authors attempted to evaluate design of leaning units with use of Learning Analytics technique on page 
flip information. Traditional formative assessment has been carried out by giving assignments and evaluating their results. 
However, the information that teacher can get from the evaluation is limited and coarse-grained. The authors set a research 
question that whether one can evaluate relation between learning objectives of learning units and the learners’ actual 
activities in the units from page flip histories. The experimental result showed that the intensity of relation between the 
assignments and learning materials were different for each unit. Quantitatively, the correlation coefficients between 
“p-value of chi-square tests between range of page flip count and grade of assignment” and “proportion of number of 
questions in an assessment with reference page” was -0.889. With use of this relation, the authors attempted to evaluate 
design of units, and found out that the thing that may be critical to get high grade in an assignment of a certain unit was how 
many times they refer to their text book, though this assignment was designed for assessment of learners’ skills to use or 
apply the learned knowledge. If teacher can get such suggestion as feedback, it is possible to utilize for formative 
assessment. 
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1. INTRODUCTION 

1.1 Digitization in Education and Learning Analytics 

In the age of paper-based learning environment, learning records were limited in respects of both quantity and 
variety. It consisted of only results of exams, grades of assignments or course histories; even they were usually 
written with pencil or ink on paper. In 1990s, with the introduction of computers into schools, learning record 
data had been moved into machine-readable form. In 2000s, Learning Management System (LMS) had been 
spread, and many types of learning activity logs were collected. Finally, in 2010s, usage of laptop or tablet PCs 
has become common in even K-12 education. It enables to record or collect various fine-grained learning 
activity logs. It is notable that these modern logs contain client-side activities, e.g. page flip, process of 
answering, eye-track, voice and environmental sound, and GPS information. In the future, even physiological 
data like blood pressure, sweating or heartbeat will be treated as “learning activity logs”. 

Along with the trend on digitization in education. Learning Analytics (LA) has become a major area in 
learning science and learning technology research. Ferguson (2012) described a definition of Learning 
Analytics as follows: “Learning analytics is the measurement, collection, analysis and reporting of data about 
learners and their contexts, for purposes of understanding and optimizing learning and the environments in 
which it occurs.” Ferguson does not refer to “report to whom” in her definition of LA. However, some target to 
report can be assumed, for example learners, teachers, or schools, and the authors think their needs in LA 
depend on each target. For example, learners want to know their own grade or position in their class, while 
teachers want to know whether their lectures worked as they had intended. 
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1.2 Learning Analytics and Formative Assessment 

In this paper, the authors focus on utilizing LA for formative assessment, in other words, LA for teachers. 
Bloom (1971) described a definition of the formative assessment as follows: “the use of systematic evaluation 
in the process of curriculum construction, teaching and learning for the purpose of improving any of these three 
processes.” This Bloom’s definition of formative assessment is very similar to Ferguson’s definition of LA, 
especially on its purpose. 

Traditional formative assessment has been carried out by giving assignment or quiz and evaluating their 
results. However, assignment or quiz is not more than once a unit and therefore information for the teachers is 
limited and coarse-grained. Therefore, teachers observe their learners carefully to get further information. 
However, it is difficult to get what they did or said, much less what they thought or what they learned. In 
cognitive science area, there are many methods to estimate such cognitive information. These methods are able 
to give fine-grained information. However, it needs qualitative assessment and it is also heavy workload to 
carry out these assessment or observation in practice. 

However, as mentioned in Section 1.1, fine granularity learning data has been recorded in 
machine-readable form. In LMS based environment, some kinds of LA are already at practical stage, and 
some LMSs have functions such as summary of login history/ course history/ grade, visualization of their week 
units, or recommendation of related courses. Educational applications also have had functions to visualize 
information of same kind. Some leading-edge LMSs or applications have functions such as evaluation of 
design of courses, units or learning materials by analyzing learning logs. That is, for example, to verify whether 
learners who take a certain course were target learners as expected, or to analyze the reason why many learners 
chose or wrote wrong answer in a certain problem. 

One factor of the learning design that the authors have focused on is learning objective. As often said, 
learning objective of some units are acquisition of knowledge or theory while that of other units are application 
of them. In Japan, Ministry of Education. Culture, Sports, Science and Technology (MEXT) has been 
conducting the National Assessment of Academic Ability (NAAA) in mathematics and language for Grades 6 
and 9 every year. According to OECD (2015), NAAA consists of two types of assessments: assessment on 
learner learning (subject knowledge; problem Type-A) and learner achievement (practical use of knowledge 
and skills; problem Type-B). Shirouzu (2015) mentioned that “problem Type-B” is roughly corresponding to 
TIMSS or PISA literacy problems respectively.” 

There are some researches or practice of utilizing learning data to assessment, and Ripley (2007 ) reported 
update on research, policy and practice, and the author of that paper called such assessment as “e-assessment”. 

1.3 Page Flip History 

Among various data in Learning Analytics, the authors have focused on “page flip” log of learning materials. 
Page flip log or page transition log is the information when and in what order the learners flipped pages, and the 
authors called the sequence of page flip log “page flip history”. This page flip history could not available from 
paper-based textbooks or other materials. In other words, teachers using paper -based textbooks conduct lessons 
on the assumption that their learners follow the teacher. However, if teachers use digital textbooks on client 
PCs, page flip history of each learner can be visualized by equipping function to collect page flip information. 

There are many researches that focused on a kind of page flip history. Nicholas (2010) reported analysis of 
transactional logs obtained from the MyiLibrary platform regarding 127 UK universities. Also, in Japan, 
Kyushu University has carried out whole-university project to collect and analyze learning data from LMS 
(Moodle) and the e-book system (BookLooper) (Kyushu University Learning Analytics Center 2016). 
Objectives of their studies are as follows: ( 1 ) improving of learning materials, (2) analyzing learning patterns, 
(3) detecting students’ comprehensive level, (4) predicting final grades, and (5) recommending e-books in 
accordance with personalization (Mouri 2016). 
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1.4 Research Question 

The purpose of this research is to verify research question that we can evaluate design of learning units from 
page flip history of the units. Especially, the authors focused on the point whether we can evaluate relation 
between learning objectives of the learning units and the learners’ actual activities in the units. If this relation 
can be visualized and evaluated, it leads to enable the quantitative formative assessment from fine granularity 
data with light workload. 


2. METHODS 

2.1 Target Courses, Units and Subjects 

The authors set two target courses. Both courses were held in Sophia University, Japan, and one of the authors 
was in charge. Target courses, units, and the number of subjects are shown in Table 1. 


Table 1. Information of Target Unit 


Course 

Unit ID 

Unit 

Date 

Number of Subjects 

Information Literacy 

IL518 

Journal search 

May 18, 2015 

71 


IL601 

Numerical data 

June 1, 2015 

64 

Learning Technology 

LT519 

Instructional Design 

May 19, 2015 

37 


LT526 

Test and Feedback 

May 26, 2015 

34 


The Information Literacy course was entry level for 1st year students, so contents were rather easy. In 
contrast. Learning Technology course was for 3rd year students, so contents were relatively difficult. 

2.2 Data Acquisition Scheme 

As previously reported (Horikoshi 2015), the authors focused on page transition of PowerPoint slide, because 
an author (Tamura) mainly uses PowerPoint in his lectures. And the other author of that paper (Yamazaki) 
developed the scheme and function to detect and transfer page flip logs automatically. This function is 
implemented in JavaScript, therefore the authors converted the original materials (Power Point files) into 
HTML and JPEG files, and added the JavaScript above. Page flip logs and other information are sent to 
Learning Record Store (LRS), when each subject changed pages. The items stored in LRS are as below: 

• Actor: student’s ID 

• Verb: lunched/ experienced 

• Page: page number 

• Date: date and times 

2.3 Procedure 

In the target classes, the author (Tamura) held lectures as usual with use of these materials. Subjects access the 
proposing materials, and page flip logs were transferred into LRS, when each subject changed pages in the 
classes. Then, learners were given assignments. 
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3. RESULT 

3.1 Page Flip History and Page Flip Count 

An example of page flip history is shown in Figure 1 . In this figure, vertical axis shows slide page number and 
horizontal axis shows time (maximum of 90 minutes), and the thick line shows a history of a teacher while 
other thin lines show subjects’. 



+me 


Figure 1. Example of Page flip history (LT526) 

“Page flip count” is the number of page transition times. In this paper, the authors define the “page flip 
count” specially as follows: transition only from a certain page to other page. Some examples of fit/unfit case 
are as below: 

■ fit case: from page3 to page4 

■ unfit case: from page3 to page3 (click a link of current page) 

3.2 Correlation between Page Flip Count and Grade 

At the beginning of analysis, the authors determined the strength of a correlation between the page flip count 
and grade of assignment of each unit, for there might be general hypothesis that learners who refer to their text 
book many times are diligent, and they are able to get high grade at assignment of that unit. Scatter plot of the 
page flip count versus grade of assignment and correlation coefficients between them are shown in Table 2 and 
Figure 2. 

Table 2. Correlation coefficients between page flip count and grade of assignment 


Unit ID 

Correlation coefficient 

p-value 

IL518 

0.122 

0.360 

IL601 

0.195 

0.142 

LT519 

-0.142 

0.409 

LT526 

-0.207 

0.263 
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Figure 2. Scatter plot of page flip count vs. grade of assignment 

As observed from Figure 2 and Table 2, there was no significant correlation between page flip count and 
grade of assignment. This result means even if learners referred to their text book many times, they cannot 
always get high grade at assignment. This is visually shown in Figure 3. Figure 3 shows page flip history of 
each learner in a unit (LT526), and the histories are ordered from upper left to lower right by their grades 
(history of the top left is the teacher’s). Blank cells show learners who were absent from the class. For example, 
the learner in the 3rd cell from the right on the top raw is the second in grade of assessment while he or she was 
absent from the class. 
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3.3 Chi-Square Test between Range of Page Flip Count and Grade 

As mentioned in Section 3.2, there was no significant correlation between page flip count and grade of 
assignment, therefore the authors verified whether there is a significant variation in grade by range of page flip 
count. The verification is performed using cross-tabulation and independent chi-squared test. For 
cross-tabulation, page flip count and learners’ grade are divided into ranges as below: 

Page flip count 

■ Range A: less than teacher’s flip 

■ Range B: more than teacher’s flip and less than twice of teacher’s 

■ Range C: more than twice of teacher’s 

Grade of assignment 

■ Range 1 : more than the first quartile 

■ Range 2: more than the median 

■ Range 3: more than the third quartile 

■ Range 4: less than the third quartile 

The result of chi-square test of independence based on the cross-tabulation is shown in Table 3. Table 3 
shows there is a significant variation in grade by range of page flip count in IL518 and IL601, but not in LT519 
and LT526. This result possibly arose from difference between two courses (IL and LT). However, there are 
still difference even in same course, between IL518 and IL601 or between LT519 and LT526. 

Table 3. Chi-Square Test between Range of Page Flip Count and Grade 


Unit ID 

r 

p-value 

IL518 

28.19 

0.000 

IL601 

14.26 

0.027 

LT519 

8.34 

0.214 

LT526 

4.58 

0.599 


3.4 Relation between Assignment and Learning Materials 

Based on the result in Section 3.3, the authors hypothesized that the intensity of relation between the 
assignments and learning materials are different for each unit, and the p-value of chi-square test in Section 3.3 
reflects this intensity. As mentioned in Section 1.2, learning objective of some units are acquisition of 
knowledge or theory while learning objective of other units are application of them. For simplicity, the authors 
referred to the former units as Type-A, and the latter units as Type-B. The assignments of the units Type-A are 
designed for assessment whether learners acquired the knowledge lectured in the class, and assignments with 
such purpose may have reference pages in the learning materials. In contrast, the assignments of the units 
Type-B are designed for assessment of learners’ skills to use or apply the learned knowledge, and assignments 
with such purpose may not have direct reference pages in the learning materials, or perhaps it may not be 
critical how many times they refer to their textbook to get high grade in assignment. 

This hypothesis consists of following two elements; (a) Whether some questions have reference page and 
the others do not, and proportion of number of questions in an assessment with reference page differ from each 
unit, and (b) Whether there is correlation between p-value of chi-square tests between range of page flip count 
and grade of assignment and proportion of number of questions in an assessment with reference page. 

First, the authors examined each assignment to verify (a), and evaluated proportion of number of questions 
in an assessment with reference page. The result of this evaluation is as follows: IL518 (20%), IL601 (10%), 
LT529 (5%), LT526 (0%). Proportion of number of questions in an assessment with reference page is given in 
parenthesis. This result demonstrated that the proportion of number of questions in an assessment with 
reference page differ from each unit, and this indicates that (a) is verified. Therefore, in second, the authors 
determined the correlation coefficients between the “p-value of chi-square tests between range of page flip 
count and grade of assignment” and the “proportion of number of questions in an assessment with reference 
page” to verify (b). Scatter plot of the p-value of chi-square tests versus the proportion is shown in Figure 4, 
and correlation coefficients between them was -0.889. It shows there is strong negative correlation between 
them, and this indicates that (b) is verified. 
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Figure 4. Scatter plot of “p-value of chi-square tests between range of page flip count and grade of assignment” and 
“proportion of number of questions in an assessment with reference page” 


4. DISCUSSION 

The result in Section 3.4 demonstrates that the intensity of relation between the assignments and learning 
materials are different for each unit. Also, Figure 6 and correlation coefficients in Section 3.4 show that larger 
proportion of number of questions with reference page the units have, more significant the p-values of 
chi-square tests are. That is, as mentioned in Section 3.4, intensity of relation between range of page flip count 
and grade of assignments reflects the intensity of relation between the assignment and learning material. Based 
on these results, it is suspected that it is possible to provide formative assessment from page flip history. In 
other words, it is possible to evaluate whether the lectures or assignments worked as teacher or designer 
intended from these two intensity of relation. 

The intensity of relation between the assignment and learning material is quantitatively evaluated as the 
“proportion of number of questions in an assessment with reference page”, and it may represent intention of 
learning design. By contrast, the intensity of relation between range of page flip count and grade of 
assignments is quantitatively evaluated as the “p-value of chi-square test between range of page flip count and 
grade of assignment”, and it may represent how the lectures or assignments worked actually. If these two 
intensity of relation are comparable with each other, this may appear as negative correlation coefficients. 

Therefore, in order to evaluate a certain unit from these correlation coefficients in terms of formative 
assessment, you should focus on whether a point of scatter plot shown in Figure 4 is on the regression line. For 
example, as observed from Figure 4, the point of IL601 is outliner. This shows that the “proportion of number 
of questions in an assessment with reference page” is small in this unit, though intensity of relation between 
range of page flip count and grade of assignments is strong. In this case, it is suggested that the thing that may 
be critical to get high grade in this assignment was how many times they refer to their text book, though an 
assignment of this unit was designed for assessment of learners’ skills to use or apply the learned knowledge. If 
teacher can get this suggestion as feedback, it is possible to utilize for formative assessment; for example, a 
teacher improves his assignment as designed for assessment of learners’ skills to use or apply the learned 
knowledge. 
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5. CONCLUSION AND FUTURE WORKS 

The result of experiment demonstrated that the intensity of relation between the assignment and learning 
materials are different for each unit. Quantitatively, the correlation coefficients between p-value of %2 tests 
versus proportion of number of questions in an assessment with reference page was -0.889. With use of this 
relation, the authors attempted evaluated design of learning units, and found out that an assignment of a certain 
unit was designed for assessment of learners’ skills to use or apply the learned knowledge. However, the thing 
that may be critical to get high grade in this assignment was how many times they refer to their text book. 

If teacher can get such suggestion as feedback, it is possible to utilize for formative assessment. It shows 
that the authors verified the hypothesis that we can evaluate design of unit from page flip history the unit, 
especially, the point whether we evaluate relation between learning objective of the unit and the learner’s actual 
activity in the unit. The author believe that it leads to enable the quantitative formative assessment from fine 
granularity data with light workload in practice. 

There are some future works related to this paper. In this paper, page flip log data was use as page flip 
counts. However, the original page flip log data contains sequence and time span. So, there is a possibility to 
analyze the data from temporal aspects. 
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